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ABSTRACT 

Tne validity of the word attack skill tests of the 
Wisconsin Design for Reading Skill Development is studied at six 
grade levels by an investigation of the relationship of these to 
several widely known reading achievement tests. The basic notion is 
that reading mastery scores derived from the diagnostic tests should 
be positively related to the achievement scores in a simple manner. 
The requirements implied by this notion are specified and the 
procedures for examination of data to determine extent of agreement 
are described. The results of the analysis indicate that the 
requirements are fulfilled in all but one case. The main conclusion 
is that the validity of the word attack subtests is supported by the 
data. (Author) 
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STUDY OF RELATIONSHIPS OF READING MASTERY LEVEL TO GENERAL 



READING ACHIEVEMENT TO VALIDATE DIAGNOSTIC READING TESTS 



Thomas J. Fischbach 
University of Wisconsin 



The Wisconsin Design for Reading Skill Development has been developed 
to aid the operation of a program of individually guided reading instruc- 
tion in grades K-6 (see Otto and Askov, 1970)* One part of the Design 
includes. a set of criterion- referenced tests of Word Attack skills at each 
of four difficulty levels* In previous work the adequacy of each of the 
individual subtests in the collection was examined by item analyses and 
determination of Hoyt internal consistency reliabilities (Fischbach, Harris 
and Quilling, 1970)* These results indicated that the subtests generally 
fulfilled the criteria that had been established for that phase of the eval 
uation* The validity of the subtesls for the purposes which led to their 
construction is the basic content validity of the items used to construct 
them. 

The present study was designed to examine the adequacy of the entire 
set of subtests in a different manner. An underlying concept of the Design 
Is the notion of a part-whole relationship between specific reading skills 
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and general reading proficiency. An implication is that general reading 
achievement can be increased by instruction directed toward development 
of specific reading skills. The subtests permit teachers to determine 
which skills their students have not mastered so that instruction may be 
appropriately directed. If this concept is valid and if the subtests are 
good measures of the important skills, then it should be possible to demon- 
strate the part-whole relationship empirically. Thus, an examination of 
the relationship between scores on the subtests and fair measures of general 
reading level should provide information concerning both the validity of the 
underlying concept and the adequacy of the subtests as measures of the 
specific skills. 

The first task was to ope utionalize this concept so that it would be 
used to guide the analysis of 'ata. This might be done in several ways. 

The way selected is one of interest because it emphasizes the dichotomy 
between mastery and non-mastery of skills. It was decided to use criteria 
already in use to determine whether a skill had been mastered, to count 
the number of skills mastered, and finally to call the result the mastery 
score. The part-whole relationship described before can be examined by 
determining what relationship, if any, this mastery score has to general 
reading levels. 

If all the skills measured bv the subtests are important for reading 
proficiency, if they ere of nearly equal importance (so that no one has 
a dominant effect), and if the relationship is a positive one, one should 
expect this relationship to take a relatively simple form. The regression 
of reading achievement scores on mastery score should take the form of a 
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"simple" polynomial function which is monotonically increasing over the 
range of possible mastery scores. The type of relationship expected would 
be graphically depicted as a smooth curve - or simply a line - which is 
predominantly linear with a positive slope which changes gradually, if at 
all, This would be a polynomial of degree two at most although this 
cutoff is somewhat arbitrary as the acceptability would depend on several 
factors. The objectives of the study are three-fold: at each level of 

difficulty to determine (1) if level of reading proficiency is related in 
any way to the number of Word Attack skills mastered; (2) if so, to de- 
termine the degree of the polynomial equation describing that relationship; 
and (3) to determine if the form of the relationship is consistent with 
that conceptualized. 



METHOD 

Data for the study were ob ained for subjects in grades one to six 
from the regular fall testing program of a school using the Wisconsin 
Design. Each grade or unit received the level of Word Attack subtests 
deemed most appropriate for initial diagnostic purposes for the children 
involved. The school supplied grade equivalent scores on two or more 
standardized reading achievement tests for each subject. Scores on the 
Metropolitan Reading Readiness Test were used for grade one. The number 
of students, the level of Word Attack, the number of Word Attack sub tests, 
and the reading achievement tests used are shown for each grade or unit in 
Table I. 
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Insert Table I about here 



Needless to say, the results of this analysis could well be specific 
to the particular school population used, to relationships existing at the 
beginning of the school year before most instruction had occurred, and to 
the particular reading achievement tests used. Moreover, the relationships 
described below cannot be regarded as more than descriptive for the data 
for this study. 

MASTERY SCORING 

The criterion for mastery on a Word Attack subtest was that 80% or more 
of the items on the subtest were answered correctly. The number of skills 
so mastered was determined and this was the mastery score for a student, 
the ’’independent" variable for the study. It is evident that the same mas- 
tery score could be obtained in several different ways. 

ANALYSIS 

The data for each level of difficulty of Word Attack was analyzed in- 
dependently of the other levels with one exception. The exception, at 
Level B, occurred because the grade three children involved were given only 
three of the four reading achievement tests given to the grade two children. 
In this case ti.e data for each grade is analyzed separately. Three grade 
groups received Level D Word Attack but each had received different forms 
of the same achievement tests. The grade effects (which were significant 
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at the .001 level on both dependant variables), which are confounded with 
form effects, were "removed" from the model before effects due to regres- 
sion. Homogeneity of regression for the three grade levels was examined 
by appropriate multivariate analysis of variance tests and the null hypoth- 
esis of no difference in the regressions among the thre-' grade groups on either 
or both the two reading achievement tests did not have to be rejected 
(p < 0.36). Except for the differences just noted and the fact that the 
number of dependent variables as well as the number of levels of mastery 
score varied among the analyses, all of the analyses followed the same pro- 
cedures and will be described collectively. All computations were done 
using the multivariate analysis of variance program by Finn (1968). 

Each mastery score defines a distinct level or "cell" in an approp- 
riate one-way layout for multivariate analysis of variance. The first 
step was to compute the standard deviations on each dependent variable for 
each of these cells. These were inspected as a check of the assumption 
of homogeneity of variance (covariances were not checked). Gross viola- 
tions of this assumption could indicate violation of the assumed equality 
of the mastery score units which, in turn, could mean a gross departure 
from the assumed approximate equality of importance of the skills. The 
results of this examination did not require rejection of the assumption of 
homogeneity of variances. 

The next step was to test the null hypothesis of no relationship be- 
tween reading achievement and mastery score on any measure of reading 
achievement by a multivariate analysis of variance test requiring signifi- 
cance at the .05 level for rejection. Only if this null hypothesis - which 
is equivalent to the hypothesis that the degree of the regression is zero - 
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were rejected, would the analysis continue. It might be noted that an 
incidental consequence of this is to ensure that the "family" or "expcri- 
inentwise" Type I error rate (see Miller, 1966) is no greater than ,05 
per analysis. 

In the next step the degree of the polynomial regression was 
determined using orthogonal polynomial contrasts and a proce lure 
adapted from one proposed by Anderson (1962) , A working as- 
sumption for this procedure is that if the polynomial has a nonzero coef- 
ficient for a term of degree k, then the polynomial is of degree k and 
all coefficients of degree k-1 or less are presu. J to be nonzero as well. 
All tests require significance at the ,05 level for rejection of null 
hypotheses. The first hypothesis tested is that coefficients of all terms 
of degree three or higher for all acheivement scores are zero. If this 
hypothesis is rejected, e.g., if the polynomial is cubic or greater order, 
the validity of the concept is tentatively rejected unless other evidence 
overrides this result. If that hypothesis is not rejected, then hypotheses 
that the quadratic and linear coefficients, respectively, are zero are 
tested in sequence to determine the degree of the polynomial (if it should 
happen that the linear coefficients ara found to be zero, the analysis is 
terminated). After this the coefficients for the best-fitting model are 
estimated. The estimated achievement scores at each mastery score are ob- 
tained and plotted in the form of curves along with actual mean values. If 
the pattern of deviations from predicted values does not suggest lack of 
fit and if the curves or lines reveal monot onically increasing relationships 
of the form expected, the validity of the concept would be said to be sup- 
ported by the data of this study. 
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RESULTS 

Significance levels actually attained for the various tests of multi- 
variate regression hypotheses are shown in Table II. In all cases the 
hypothesis of no relationship must be rejected. In four of the five cases 
the degree of the polynomial was found to be two 01 less, and in three of 
the five cases a linear model was found to be adequate. The graphs of 
best fitting models and observed values are shown in Figures 1 to 5. 



Insert Table II about here 



Observed and predicted mean reading achievemen t score at each mastery 
score for each reading achievement test and each level of difficulty are 
shov;n in Table III. These are the values plotted in Figures 1-5. 



Insert Table III about here 



The one case where a cubic or higher order polynomial appears neces- 
sary is grade 2 students on Level B Word Attack. Interestingly, a linear 
model was found to be fairly adequate for grade 3 students on Level B 
Word Attack, although the number of students - and hence the power of 
tests - was less in the latter case. In many respects the relationships, 
when plotted as In Figures 2 and 3, appeared remarkably similar for both 
groups. The largest coefficients in both cases are the positive ones for 
the linear terms and reading achievement does tend to increase with 
mastery level. Beea iso the quadratic and cubic coefficients are not large 

in the grade 2 case the graph in Figure 2 shows the best fitting lines 
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rather than curves. When viewing these graphs one should note that the 
deviations from the predicted values are statistically signif ican t . Some 
of the larger deviations occur for the four highest mastery scores where 
the numbers of subjects are small (see Table III for actual values). 

When the reading tests were examined individually using univariate 
F-ratios it was found that the deviations from second degree polynomial 
regression were large enough to be statistically significant for Word 
Meaning (p < .011) and Paragraph Meaning (p < .005; but not for Vocabulary 
(p < .09) or Wo r d Study (p < .81). However, this is not strong evidence 
of good fit to a second or first degree polynomials for the latter two 
cases. But if the analysis had been continued in these cases, individually, 
the hypotheses that the quadratic coefficients are zero would not be re- 
jected (p < .27 and p < .72, respectively) but the comparable hypotheses 
about linear coefficients would be (p < .001 in both cases) . 

A second-degree polynomial is necessary for Level C Word Attack for 
Grade 3 students. The relationships of mastery score to the three de- 
pendent variables was found to be mono tonically increasing with one minor 
exception. The exception occurs for Word Study where the predicted value 
decreases from 1.88 at mastery score 0 to 1.85 at 1 but the predicted values 
increase with mastery score from that point on. The graphs in Figure 4 
and the predicted values for this case in Table III dc presume that the 
regression is that of a second degree polynomial for all three dependent 
variables. Additional analysis reveals that if i .a hypothesis that the 
coefficient of the second degree term is zero had been tested for each de- 
pendent variable, this hypothesis would be rejected only in the case of 
Word Study (p < .01) but not for Word Meaning (p < .054) or for Paragraph 
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Meaning (p < .17) « However , this does not preclude the possibility that 
some linear combination of the second degree coefficients for the latter 
two variables : s non-zero. 

A linear model was found to be adequate for Level A for Grade 1 stu- 
dents, As noted in Table II the null hypothesis that the linear coeffi- 
cients for all six Metropolitan Readiness scores are zero must be rejected. 
Separate univariate analyses indicate positive coefficients for five sub- 
tests 'p < .02 in all cases). For one, Word Meaning, the hypothesis that 
the coefficient is zero cannot be rejected (p < .41) although the esti- 
mated coefficient is positive in that case, too. 

The best fit to linear regression was found at Level D Word Attack 
for grades 4, 5, and 6. Moreover, the slopes of the best fitting lines 
were steepest in this case. 



DISCUSSION 



The results of this analysis generally provide support for a basic 
notion of the Wisconsin Design that general reading level is related to 
the number of specific reading skills mastered. However, the results here 
must be viewed as tentative and should be interpreted in the context of 
the limitations of the data used. One interesting result, which may of 
course be the result of unknown factors, is the tendency for the expected 
relationship to be supported more strongly at the higher grade levels. 

This nay reflect the cumulative results of previous instruction or that 
reliable measurement of general reading achievement is more difficult for 
less advanced students. 
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These results do not ’’prove" the validity of the Word Attack subtests 
nor the practical utility of the Design. Nor could this be expected with 
any study of this 'ype. The basic test of the validity of the subtests 
is the content validity which was established before this study, and the 
utility of the Design is not-; being tested in a carefully planned study 
for that purpose. However, the practical import of this study is that 
with relatively low cost it was possible to check for gross deficiencies 
and thus avoid the needless expense of larg^ scale studies in the event 
of negative findings. 
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Table I Number of Subjects and Tests Used by Grade 



Number 

Grade Students 



Word Attack Sub tests 
Difficulty Level Number 



Reading Achievement Tests 



1 


68 


A 


6 


Metropolitan Reading Tests, Form A 
(Word Meaning, Listening, Matching 
Alphabetizing, and Copying) 


2 


56 


B 


11 


Stanford Achievement Test Primary 
1, Form W: (Word Meaning, Para- 

graph Meaning; Vocabulary, Word 
Study Skills) 


3 


31 


B 


11 


Stanford Achievement Test Primary 
1, Fcrm W: (Word Reading, Para- 

graph Meaning, Word Study) 


3 


50 


C 


13 


Stanford Primary 2 Form W: (Word 

Meaning, Paragraph Meaning, and 
Word Study Skills) 


rotal 4-6 

4 

5 

6 


227 

81 

82 

64 


D 


6 


Stanford Achievement 

Form W Intermediate 1 

Form X Intermediate 1 

Form W Intermediate 2 

(Word Meanings and Word Study 

Skills) 
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Table II Summary of Results of Analysis of Variance for Regression 
of Reading Achievement Scores on Mastery Score by Word 
Attack Level 



Source of 
Effect 


A 


Significance Levels Attained 
Word Attack Level and Grade 

B 0 


D 


Grade 1 


2 3 


3 


4-6 


Between Mastery 
Scores — All com- 
ponents 


.002 


.001 .02 


.001 


.001 


By Component 
Linear 


.002* 


.001 .001* 


.001 


.001* 


Quadratic 


.17 


.09 .27 


.02* 


.90 


Cubic & all higher 


.60 


.001* 82 


.44 


.19 


Number of dependent 
variables 


5 


4 3 


3 


2 


Degrees of freedom 
per dependent vari- 
able 


61 


46 21 


37 


206** 


Number of non-empty 
cells 


7 


10*** 10*** 


13*** 


7 


Number possible cells 


7 


12 12 


14 


7 



* Indicates significance level of last hypothesis tested. Those appearing 
above these, are for reference only. 

** 2 degrees of freedom used for grade effects and 12 for grade x mastery level 
effects . 

*** Cells for which there were no subjects: 

Level B Grade 2-2 empty cells were at tv/o highest possible scores 
Level B Grade 3-2 empty cells: one at lowest possible score and one 

at highest possible score 

Level C Grade 3-1 empty cell at score 11 with score 12 as the highest 
possible score. 
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Table III Predicted and Observed Mean reading Achievement Scores by Mastery Score- 
by Word Attack Level of Difficulty 

Level A for Grade 1 Students - These values are plotted in Figure 1 



Metropolitan Reading Readiness Subtests 



Mastery 

Score 



No. of 
Sub j . 



Word Meaning 
Ob s . P re d . 



Listening 
Obs. Pred. 



Matching 
Obs . Pred. 



Alphabetizing 
Obs. Pred. 



Copying 
Obs. Pred. 



0 

1 

2 

3 

4 

5 

6 

Total 



2 

3 

6 

8 

19 

19 

11 

68 



Std. Deviation 

Estimated 
Constant, p 
Slope, & 
Stan. Error 

of a 



8.00 

10.67 

10 .00 

7.62 

9.05 

8.84 

11.00 

9.26 

3.13 



8.42 

8.63 

8.84 

9.04 

9.25 

9.46 

9.67 



8.42 

0.21 

0.25 



4.50 

9.67 

8.67 
8.38 

[ 10 . 32 

9.53 

[ 10.73 

9.59 

2.82 



7.35 

7.90 

8,45 

9.01 

9,56 

10.11 

10,66 



7.35 

0.55 

0.23 



3.00 

4.33 
6.83 
6.88 
6.42 
8.47 

10.36 

7.53 

3.34 



3.34 

4.37 

5.40 

6.44 

7.47 

8.50 

9.54 



3.34 

1.03 

0.27 



1.00 

8 . 33 

5.67 

7.63 

10.16 

11.26 

13.64 

9.99 

3.84 



3.07 

4.77 

6.48 

8.18 

9.89 

11.59 

13.29 



3.07 

1.70 

0.31 



0.50 
1. 33 

2.67 
5.25 

6.42 

5.68 
6.91 

5.43 
3.81 



1.51 

2.48 

3.44 

4.41 

5.37 

6.33 

7.30 



1.51 

0.96 

0.31 



Each score is the mean number of items correct observed or predicted for the mastery 
score group. Predicted scores, y(x), are obtained froid the equation y(x) = p + 3x 
using the estimated values given and where x is the mastery score. 
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Le vel B for Gr ade 2 Stu dents - These values are plotted in Figure 2 



Stanford Achievement Test Primary 1 



Mastery 

Score 


1 No. of 
Subjects 


Word Meaning 
Obs. Pred. 


Paragraph Meaning 
Obs. Pred. 


Vocabulary 
Obs. Pred. 


Word Study 
Skills 

Obs. Pred. 


0 


11 


1.34 


1.31 


1.47 


1.32 


1.54 


1.56 


lc 45 


1.42 


1 


12 


1.53 


1.49 


1.47 


1.48 


1.71 


1.76 


1.59 


1 .65 


2 


5 


1.70 


1.66 


1.64 


1.63 


1.80 


1.96 


1.86 


1.87 


3 


10 


1.80 


1 . S3 


1.67 


1.79 


2.32 


2.16 


2.15 


2.10 


4 


5 


1.86 


2.00 


1.68 


1.94 


2.24 


2.36 


2.40 


2.32 


5 


4 


1,85 


2.17 


1.83 


2.10 


3.10 


2.55 


2.33 


2.55 


6 


4 


2.48 


2.34 


2.30 


2.26 


2.38 


2.75 


3.00 


2.78 


7 


1 


3.60 


2.52 


3.10 


2.41 


4.40 


2,95 


2.80 


3.00 


8 


3 


2.50 


2.68 


3.23 


2.57 


2.80 


3.15 


3.33 


3.23 


9 


1 


3.20 


2.86 


1.90 


<s» 

CM 


2.90 


3.35 


3.00 


3.45 


10 


— 


— 


3.02 


— 


2.88 


— 


3.54 


— 


3.68 


11 


— 


— 


3.18 


— 


3.03 


— 


3.74 




3.90 


Total ' 


56 


1.79 


— 


1.75 


— 


2.11 


— 


2.05 





Standard Deviation 


0.30 


— 


0.34 


— 


0.58 


— 


0.45 


— 


Estimated 




















Constant, p 


— 


1.31 


— 


1.32 


— 


1.56 


— 


1.42 


Slope, $ 


— 


0.172 


— 


0.156 


— 


0.198 


— 


0.226 


S tan. Error, 8 


— 


0.016 


"" 


0.018 


— 


0.032 


— 


0.025 



Each score is the mean grade equivalent ocore observed or predicted for the mas- 
tery score group. Predicted scores, y(>;) can be obtained from the equation: 

y(x) - \i + 8x using the estimated parameters given and where x is 

the mastery score. Deviations from these predicted values are statistically sig- 
nificant. 
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Level B for Gr a de 3 Students - These values are plotted in Figure 3 



Stanford Achievement Test Primary 1 




Each score is the mean grade equivalent score observed or predicted for the mas- 
tery score group. Predicted scores, y(x), are obtained from the equation 

A . *> * 

y(x) e \i + 3x using the estimated parameters given and where x is the 



0 



mastery score. 
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L evel C for Gr ade 3 - These values are plotted in Vigure 4. 



Stanford Achievement Test 



Mastery 

Score 


No. of 
Subjects 


Word Meaning 
Obs. Pred. 




' Paragraph Meaning 
Obs. Pred. 


Word Study Skills 
Obs. Pred. 


0 


4 


2.05 


2.04 




1.98 


| 1.85 


1.85 


1.88 


1 


3 


2.27 


2.07 




1.70 


1.95 


1.90 


1.85 


2 


9 


2.20 


2.13 




2.22 


2.06 


2.00 


1.88 


3 


5 


2.28 


2.22 




2.24 


2.20 1 


1.90 


1.96 


4 


3 


1.83 


2.32 




2.00 


2.36 


1.97 


2.11 


5 


4 


1.80 


2.46 




2.10 


2.54 


2.02 


2.30 


6 


3 


2.77 


2.61 




2.40 


2.73 


2.43 


2.56 


7 


4 


3.08 


2.80 




3.18 


2.95 


2.40 


2.87 


*■> 

c 


5 


3.06 


3.00 




3.36 


3.18 


3.92 


3.24 


9 


2 


3.70 


3.24 




3.85 


3.44 


3.30 


3.66 


10 


4 


3.78 


3.49 




4.00 


3.71 


4.48 


4.14 


11 

12 


0 

4 


3.75 


4.08 




3.981 


4.31 


f.OO 


5.27 


Total 


50 


2.63 


— 




2.70 





2.71 





Standard Deviation 


0.49 


— 




0.58 


— 


0.91 


— 


Estimated 

Constant 






2.04 






1.85 




1.88 


Linear Coef. 
Quadratic Coef. &2 


— 


0.021 




— 


0.089 


— 


-0.055 


— 


0.012 




— 


0.010 


— 


0.028 


Stan. Error, Si 


— 


0.071 




— 


0.085 


— 


0.133 


Stan. Error, S 2 


— 


0.006 




— 


0.0 07 


— 


0.011 



Each score is mean grade equivalant score observed or predicted for the mastery score 
observed or predicted for the mastery score group. Predicted score, y(x), are obtained 
from the equation y(x) * u + using the estimated coefficients given where 

x Is the mastery score. 
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Level D for Grades 4, 5, and 6 - These values are plotted in Figure 5. 

















Stanford Achieve: 


:cent Test 






Mastery 

Score 




No. of 
Subjects 
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Observed scores are observed moan grade equivalent scores corrected by the 
factors given in the rov "deviations from U by grade" to remove grade effects. Pre- 
dicted values, y(x), are given by the equation y(x) - p + Sx where the estimates 
of parameters are those given and x is mastery score. 
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Metropolitan Readiness Mean Scores: Predicted and 

Observed by Mastery Score by Subtest for Grade 1 Students 
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FIGURE 3: 



Stanford Achievement Test Primary Is Predicted and Observjd 
by Mastery Score by Sub test for Crude 3 Students 
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FIGURE 4: Stanford Achievenent Test Mean Scores: Predicted and Observed 

by Mastery Score by Subtest for Grade 3 Students 
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FIGURE 5: Stanford Achleverant 
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